Skip to content

feat(clickhouse): add directory-based TLS certificate loading#3938

Merged
whites11 merged 2 commits into
mainfrom
kk/clickhouse-peer-tls-directory-approach
May 5, 2026
Merged

feat(clickhouse): add directory-based TLS certificate loading#3938
whites11 merged 2 commits into
mainfrom
kk/clickhouse-peer-tls-directory-approach

Conversation

@jkaflik

@jkaflik jkaflik commented Feb 16, 2026 •
edited by whites11
Loading

Copy link
Copy Markdown
Contributor

Add tls_certificate_directory option to ClickHouse peers for loading TLS certificates from a directory (e.g. a Kubernetes Secret mounted by cert-manager).

  • Client certs are re-read on every TLS handshake via GetClientCertificate, picking up rotations without restart
  • CA cert is loaded once at connection time
  • Falls back to existing inline cert fields when directory is not set
  • Fix POSIX compatibility bug in generate-protos.sh (&> bashism broke proto generation on cache miss)

@jkaflik jkaflik changed the base branch from kk/clickhouse-peer-tls-k8s-secret to main February 16, 2026 11:47
@CLAassistant

CLAassistant commented Feb 16, 2026 •
edited
Loading

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@codecov

codecov Bot commented Feb 16, 2026 •
edited
Loading

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
2208 2 2206 189
View the top 2 failed test(s) by shortest run time 
github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuitePG_CH
Stack Traces | 0.01s run time 
=== RUN   TestPeerFlowE2ETestSuitePG_CH
=== PAUSE TestPeerFlowE2ETestSuitePG_CH
=== CONT  TestPeerFlowE2ETestSuitePG_CH
--- FAIL: TestPeerFlowE2ETestSuitePG_CH (0.01s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuitePG_CH/Test_Binary_Format_Raw
Stack Traces | 133s run time 
=== RUN   TestPeerFlowE2ETestSuitePG_CH/Test_Binary_Format_Raw
=== PAUSE TestPeerFlowE2ETestSuitePG_CH/Test_Binary_Format_Raw
=== CONT  TestPeerFlowE2ETestSuitePG_CH/Test_Binary_Format_Raw
2026/05/05 07:49:54 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/05/05 07:49:54 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/05/05 07:49:54 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id"
2026/05/05 07:49:54 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id"
2026/05/05 07:49:54 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_15409950964551334889 CURSOR FOR SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id" args=[]
2026/05/05 07:49:54 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id" channelLen=0
2026/05/05 07:49:54 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_15409950964551334889
2026/05/05 07:49:54 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_15409950964551334889 records=4 bytes=40 channelLen=3
2026/05/05 07:49:54 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id" rows=4 bytes=40 channelLen=3
2026/05/05 07:49:54 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_15409950964551334889
2026/05/05 07:49:54 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_15409950964551334889 records=0 bytes=0 channelLen=0
2026/05/05 07:49:54 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/05/05 07:49:54 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/05/05 07:49:54 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id" rows=4 bytes=40 channelLen=0
    clickhouse_test.go:1064: WaitFor waiting for CDC count 2026-05-05 07:49:58.523131363 +0000 UTC m=+558.726217811
    clickhouse_test.go:1068: WaitFor waiting for CDC count 2026-05-05 07:49:58.528609761 +0000 UTC m=+558.731696208
2026/05/05 07:49:58 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id"
2026/05/05 07:49:58 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id"
2026/05/05 07:49:58 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_17858405722765214869 CURSOR FOR SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id" args=[]
2026/05/05 07:49:58 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id" channelLen=0
2026/05/05 07:49:58 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_17858405722765214869
2026/05/05 07:49:58 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_17858405722765214869 records=4 bytes=40 channelLen=3
2026/05/05 07:49:58 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id" rows=4 bytes=40 channelLen=3
2026/05/05 07:49:58 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_17858405722765214869
2026/05/05 07:49:58 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_17858405722765214869 records=0 bytes=0 channelLen=0
2026/05/05 07:49:58 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/05/05 07:49:58 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/05/05 07:49:58 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_8rosou3j.\"test_update_pkey_chunking_initial_load_enabled\" ORDER BY id" rows=4 bytes=40 channelLen=0
    clickhouse.go:97: 
        	Error Trace:	.../flow/e2e/congen.go:44
        	            				.../flow/e2e/clickhouse.go:97
        	            				.../flow/e2e/clickhouse.go:79
        	            				.../flow/e2e/clickhouse.go:154
        	            				.../flow/e2e/test_utils.go:240
        	            				.../flow/e2e/test_utils.go:852
        	            				.../flow/e2e/test_utils.go:237
        	            				.../flow/e2e/clickhouse_test.go:1068
        	            				.../flow/e2e/clickhouse_test.go:1083
        	Error:      	Received unexpected error:
        	            	unable to establish connection with catalog: FATAL: terminating connection due to administrator command (SQLSTATE 57P01)
        	Test:       	TestPeerFlowE2ETestSuitePG_CH/Test_Binary_Format_Raw
    clickhouse.go:107: begin tearing down postgres schema pgch_0ontqtkm
2026/05/05 07:50:07 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_zyyp9oxs.\"test_nullengine\" ORDER BY id"
2026/05/05 07:50:07 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_zyyp9oxs.\"test_nullengine\" ORDER BY id"
2026/05/05 07:50:07 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_17726626514862212290 CURSOR FOR SELECT id,\"key\" FROM e2e_test_pgch_zyyp9oxs.\"test_nullengine\" ORDER BY id" args=[]
2026/05/05 07:50:07 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_zyyp9oxs.\"test_nullengine\" ORDER BY id" channelLen=0
2026/05/05 07:50:07 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_17726626514862212290
2026/05/05 07:50:07 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_17726626514862212290 records=3 bytes=26 channelLen=2
2026/05/05 07:50:07 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_zyyp9oxs.\"test_nullengine\" ORDER BY id" rows=3 bytes=26 channelLen=2
2026/05/05 07:50:07 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_17726626514862212290
2026/05/05 07:50:07 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_17726626514862212290 records=0 bytes=0 channelLen=0
2026/05/05 07:50:07 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_zyyp9oxs.\"test_nullengine\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/05/05 07:50:07 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/05/05 07:50:07 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_zyyp9oxs.\"test_nullengine\" ORDER BY id" rows=3 bytes=26 channelLen=0
    clickhouse.go:107: 
        	Error Trace:	.../flow/e2e/pg.go:147
        	            				.../flow/e2e/clickhouse.go:107
        	            				.../flow/e2eshared/e2eshared.go:38
        	            				.../hostedtoolcache/go/1.26.2.../src/testing/testing.go:1317
        	            				.../hostedtoolcache/go/1.26.2.../src/testing/testing.go:1667
        	            				.../hostedtoolcache/go/1.26.2.../src/testing/testing.go:2030
        	            				.../hostedtoolcache/go/1.26.2.../src/runtime/panic.go:694
        	            				.../hostedtoolcache/go/1.26.2.../src/testing/testing.go:1022
        	            				.../flow/e2e/congen.go:44
        	            				.../flow/e2e/clickhouse.go:97
        	            				.../flow/e2e/clickhouse.go:79
        	            				.../flow/e2e/clickhouse.go:154
        	            				.../flow/e2e/test_utils.go:240
        	            				.../flow/e2e/test_utils.go:852
        	            				.../flow/e2e/test_utils.go:237
        	            				.../flow/e2e/clickhouse_test.go:1068
        	            				.../flow/e2e/clickhouse_test.go:1083
        	Error:      	failed to teardown postgres schema
        	Test:       	TestPeerFlowE2ETestSuitePG_CH/Test_Binary_Format_Raw
        	Messages:   	pgch_0ontqtkm: failed to drop replication slots: ERROR: replication slot "peerflow_slot_ch_binary_format_raw_pgch_0ontqtkm" is active for PID 27012 (SQLSTATE 55006)
--- FAIL: TestPeerFlowE2ETestSuitePG_CH/Test_Binary_Format_Raw (133.35s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@whites11 whites11 force-pushed the kk/clickhouse-peer-tls-directory-approach branch from a7796e9 to f50a871 Compare April 30, 2026 05:14
@whites11 whites11 marked this pull request as ready for review April 30, 2026 05:14
@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

❌ Test Failure

Analysis: Real build failure: generated proto types (FlowOperation, PostgresConfig, DynamicSetting, etc.) are undefined after the "Stubbed generated modules" commit, causing compilation to fail before any tests run.
Confidence: 0.97

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

@jkaflik jkaflik requested review from ilidemi and jgao54 April 30, 2026 05:22
@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

❌ Test Failure

Analysis: Deterministic build failure caused by undefined protobuf types (protos.FlowOperation, protos.PostgresConfig, etc.) after generated modules were stubbed out in the latest dependency update commit.
Confidence: 0.99

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

❌ Test Failure

Analysis: Real build failure: generated proto stubs from commit #4250 left protos.FlowOperation, protos.PostgresConfig, protos.DynamicSetting, and other symbols undefined, causing a compile error before any tests ran.
Confidence: 0.97

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

❌ Test Failure

Analysis: Deterministic build failure: proto-generated symbols (protos.FlowOperation, protos.PostgresConfig, protos.DynamicSetting, etc.) are undefined in internal/context.go, internal/catalog.go, and internal/dynamicconf.go, almost certainly caused by the most recent commit "Stubbed generated modules (#4250)" which stubbed out the generated proto code.
Confidence: 0.97

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

@claude

claude Bot commented Apr 30, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@whites11 whites11 force-pushed the kk/clickhouse-peer-tls-directory-approach branch from 4a3052e to c92cd39 Compare April 30, 2026 15:02
@whites11 @jkaflik @claude
feat(clickhouse): load TLS certificates from directory for cert-manag…
29f3711 
…er support

Add TLS certificate directory support for ClickHouse peers, enabling
certificates to be loaded from a mounted Kubernetes Secret (cert-manager).

Client certs are re-read on every TLS handshake via GetClientCertificate,
so rotated certificates are picked up without requiring a restart.

Also fixes generate-protos.sh to use POSIX-compatible shell syntax
(the &> bashism silently broke proto generation on cache miss).

Co-Authored-By: Kuba Kaflik <kuba.kaflik@clickhouse.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@whites11 whites11 force-pushed the kk/clickhouse-peer-tls-directory-approach branch from c92cd39 to 29f3711 Compare April 30, 2026 15:05
@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

❌ Test Failure

Analysis: Build failure caused by shared.Ptr being undefined in e2e/clickhouse_tls_test.go:36 — a real code defect where a symbol was removed from the shared package but the test file still references it.
Confidence: 0.99

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: All three failures across different matrix configurations share the same transient FATAL: terminating connection due to administrator command (SQLSTATE 57P01) error, indicating the PostgreSQL catalog server killed connections mid-test under high concurrency (-p 32), not a code defect.
Confidence: 0.95

✅ Automatically retrying the workflow

View workflow run

@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: The test failed because the PostgreSQL catalog connection was killed by an administrator command (SQLSTATE 57P01 / admin_shutdown), which is an infrastructure timing issue caused by another test's teardown, not a code defect.
Confidence: 0.92

✅ Automatically retrying the workflow

View workflow run

@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: The test TestApiPg/TestTableAdditionWithoutInitialLoad failed with FATAL: terminating connection due to administrator command (SQLSTATE 57P01), a PostgreSQL error indicating the catalog DB connection was externally killed mid-test — a transient infrastructure issue, not a code bug.
Confidence: 0.95

✅ Automatically retrying the workflow

View workflow run

@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: The test failed due to SQLSTATE 57P01 ("terminating connection due to administrator command"), a transient PostgreSQL catalog connection drop caused by CI infrastructure, not a code defect.
Confidence: 0.95

✅ Automatically retrying the workflow

View workflow run

@whites11 whites11 changed the title feat(clickhouse): another approach for TLS certificate from secret feat(clickhouse): add directory-based TLS certificate loading Apr 30, 2026
@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: Test failed with PostgreSQL SQLSTATE 57P01 (admin_shutdown) — the catalog connection was forcibly terminated by another parallel test, almost certainly a ToxiProxy-based test disrupting the shared postgres catalog used by TestApiMongo/TestTableAdditionWithoutInitialLoad.
Confidence: 0.92

✅ Automatically retrying the workflow

View workflow run

@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: Three MySQL→ClickHouse e2e tests timed out waiting for CDC/initial load data via WaitFor polling loops, with the entire suite running 863s out of a 900s limit — a classic timing/resource-contention flake, not a logic regression.
Confidence: 0.85

✅ Automatically retrying the workflow

View workflow run

ilidemi
ilidemi approved these changes May 5, 2026
View reviewed changes
Comment thread flow/connectors/clickhouse/clickhouse.go
@whites11
fix(clickhouse): error on unexpected CA cert read failures in directo…
901bad0 
…ry TLS

Distinguish "file not found" (ca.crt is optional) from other read errors
(e.g. permission denied) that would otherwise be silently swallowed,
leading to confusing TLS failures downstream.
@github-actions

github-actions Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Flaky Test Detected

Analysis: TestPeerFlowE2ETestSuitePG_CH/Test_Binary_Format_Raw failed during teardown due to a race condition where the replication slot was still held by an active PID when cleanup tried to drop it (SQLSTATE 55006), a timing-dependent failure in a high-parallelism (-p 32) e2e test environment.
Confidence: 0.85

✅ Automatically retrying the workflow

View workflow run

@whites11 whites11 merged commit 20728e8 into main May 5, 2026
26 of 32 checks passed
@whites11 whites11 deleted the kk/clickhouse-peer-tls-directory-approach branch May 5, 2026 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@whites11 whites11 whites11 left review comments
@ilidemi ilidemi ilidemi approved these changes
@jgao54 jgao54 Awaiting requested review from jgao54

Assignees

@whites11 whites11

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jkaflik @CLAassistant @whites11 @ilidemi